Copyright © 1992, 1993 Free Software Foundation, Inc.
Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.
The Appendix “Tool Command Language Overview” was written by John Ousterhout, of the University of California at Berkeley. Special copyright terms apply to that Appendix; @pxref{Tcl,,Tool Command Language Overview}.
Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided also that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.
Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Beyond this, DejaGnu offers several advantages for testing:
DejaGnu is written in expect
, which in turn uses Tcl (Tool
command language).
Running tests requires two things: the testing framework, and the test
suites themselves. Tests are usually written in expect
using
Tcl, but you can also use a Tcl script to run a test suite that is not
based on expect
.
1.1 Running existing tests | A first look at running DejaGnu tests | |
1.2 What does a DejaGnu test look like? | What does a DejaGnu test case look like? | |
1.3 Design goals | Goals behind DejaGnu | |
1.4 Future directions | Where is DejaGnu going? | |
1.5 Tcl and Expect | Reading more about Tcl and Expect |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
To run tests from an existing collection, first use configure
as
usual to set up the source directory containing the tests. Then try
running
make check
If the check
target exists, it usually saves you some
trouble—for instance, it can set up any auxiliary programs or other
files needed by the tests.
Once you have run ‘make check’ to build any auxiliary files, you
might want to call the test driver runtest
directly to repeat the
tests. You may also have to call runtest
directly for test
collections with no check
target in the ‘Makefile’.
Typically, you must use two command-line options: ‘--tool’, to specify which set of tests to run(1), and ‘--srcdir’, to specify where to find test directories.
For example, if the directory ‘gdb/testsuite’ contains a collection of DejaGnu tests for GDB, you can run them like this:
eg$ cd gdb/testsuite eg$ runtest --tool gdb Test output follows, ending with: === gdb Summary === # of expected passes 508 # of expected failures 103 # of unexpected successes 0 # of unexpected failures 0 /usr/latest/bin/gdb version 4.8.3 -nx
You can use the option ‘--srcdir’ to point to some other directory containing a collection of tests:
eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite
These examples assume a native configuration, where the same
computer runs both runtest
and the tests themselves. When you
have a cross configuration, the tests run on a different computer,
controlled by the host running runtest
. In this situation, you
need the option ‘--name’ to specify the network address for the
other computer:
eg$ runtest --tool gdb --name vx9.munist.com
If you always use the same option values, you can record them in a file called ‘site.exp’, rather than typing them each time. See section Site Configuration.
By default, runtest
prints only the names of the tests it runs,
the tests that fail (or succeed unexpectedly), and a summary. To
include all tests in the output (those that pass, and those that are
expected to fail), use the ‘--all’ flag. For more verbose output
about processes being run, communication, and so on, use
‘--verbose’. To see even more output, use multiple ‘--verbose’
options. See section Using runtest
, for a more
detailed explanation of each runtest
option.
Test output goes into two files in your current directory: summary output in ‘tool.sum’, and detailed output in ‘tool.log’. (tool refers to the collection of tests; for example, after a run with ‘--tool gdb’, look for output files ‘gdb.sum’ and ‘gdb.log’.) See section The files DejaGnu writes.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Each DejaGnu test is an expect
script; the tests vary widely in
complexity, depending on the nature of the tool and the feature tested.
Here is a very simple GDB test—one of the simplest tests shipped with DejaGnu (extracted from ‘gdb.t00/echo.exp’):(2)
# send a string to the GDB stdin: send "echo Hello world!\n" # inspect the GDB stdout for the correct reply, # and determine whether the test passes or fails: expect { -re "Hello world.*$prompt $" { pass "Echo test" } -re "$prompt $" { fail "Echo test" } timeout { fail "(timeout) Echo test" } } |
Though brief, this example is a complete test. It illustrates some of the main features of DejaGnu test scripts:
send
(to give it commands) and expect
(to analyze its responses).
expect
command uses a list of pairs; a pattern (regular
expression if ‘-re’ specified), followed by an action to run if the
pattern matches output from the program. Only the action for the
first matching pattern will execute.
pass
and fail
to record the
test outcome.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
DejaGnu grew out of the internal needs of Cygnus Support. Cygnus maintains and enhances a variety of free programs on many different environments, and we needed a testing tool that:
DejaGnu was designed to meet these goals.
Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to test the GDB back end on several remote targets.
Probably the greatest challenge was testing in a cross-development
environment (which can be a real nightmare). Most cross-development
environments are customized by each developer. Even when buying
packaged boards from vendors there are many differences. The
communcation interfaces vary from a serial line to ethernet. DejaGnu
was designed with a modular communication setup, so that each kind of
communication can be added as required, and supported thereafter. Once
a communication procedure is coded, any test can use it. Currently
DejaGnu can use rsh
, rlogin
, telnet
, tip
,
kermit
, and mondfe
(3) for remote
communications.
Julia Menapace first coined the term “Deja Gnu” to describe an earlier testing framework at Cygnus Support. When we replaced it with the Expect-based framework, we could not resist using the name DejaGnu all over again…
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
In the near future, there are two parallel directions for DejaGnu
development. The first is to add support for more targets; part of this
consists of porting Tcl and expect
to new hosts as well. The
second is to add more test cases, and support for more tools. It would
be particularly useful to add tests for conformance to the ANSI C
and C++ specifications, to POSIX, and to other standards.
Another useful extension would be to permit testing programs with a more
complex interface, whether text based or GUI based. Two components
already exist: a Tcl based X window toolkit, and a terminal package for
expect
. Both of these must be merged into DejaGnu in a way that
permits testing programs that run in each environment.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter Usenix conference, Tcl: An Embeddable Command Language. That paper is included in PostScript form in the ‘doc’ subdirectory of the Tcl distribution. @xref{Tcl,,Tool Command Language Overview}, for details on Tcl.
Don Libes introduced expect
in his paper expect: Curing
Those Uncontrollable Fits of Interaction at the 1990 Summer Usenix
conference. The paper is included in PostScript form in the
expect
distribution (as are several other papers about
expect
). @xref{Expect,,Expect: Programmed Dialogue}, for details
on expect
.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
runtest
runtest
is the executable test driver for DejaGnu. You can
specify two kinds of things on the runtest
command line: command
line options, and Tcl variables for the test scripts. The options are
listed alphabetically below. You can control the defaults for all
runtest
options; see Site configuration.
runtest
returns an exit code of 1
if any test
has an unexpected result; otherwise (if all tests pass or fail as
expected) it returns 0
as the exit code.
runtest
flags the outcome of each test as one of these four cases:
PASS
The most desirable outcome: the test succeeded, and was expected to succeed.
XPASS
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
FAIL
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.
XFAIL
A test failed, but it was expected to fail. This result indicates no change in a known bug.
runtest
may also display the following messages:
ERROR
Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target.
WARNING
Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.
This is the full set of command line options that expect
recognizes.
runtest envvar=value… [ --all ] [ --connect type ] [ --debug ] [ --diff ] [ --help ] [ --mail "name1 name2 …" ] [ --noreboot ] [ --outdir path ] [ --objdir path ] [ --runtests "file1 file2…" ] [ --srcdir path ] [ --strace n ] [ --name name ] --tool tool [ --target string ] [ --host string ] [ -v | --verbose ] [ -V | -version ]
tclvar=value
You can define Tcl variables for use by your test scripts in the same
style used with make
for environment variables. For example,
‘runtest GDB=gdb.old’ defines a variable called ‘GDB’; when
your scripts refer to ‘$GDB’ in this run, they use the value
‘gdb.old’.
The default Tcl variables used for most tools are defined in the
main DejaGnu Makefile
; their values are captured in the
‘site.exp’ file. See section Site configuration.
--all
Display all test output. By default, runtest
shows only the
output of tests that produce unexpected results; that is, tests with
status ‘FAIL’ (unexpected failure), ‘XPASS’ (unexpected
success), or ‘ERROR’ (a severe error in the test case itself).
Specify ‘--all’ to see output for tests with status ‘PASS’
(success, as expected) ‘XFAIL’ (failure, as expected), or
‘WARNING’ (minor error in the test case itself).
--baud baud rate
-b
Set the default baud rate to something other than 9600. (Some serial
interface programs, like tip
, use a separate initialization file
instead of this value.)
--connect type
Connect to a target testing environment as specified by type, if
the target is not the computer running runtest
. For example, use
‘--connect’ to change the program used to connect to a “bare
board” boot monitor. The choices for type in the DejaGnu 1.0
distribution are ‘rlogin’, ‘telnet’, ‘rsh’, ‘tip’,
‘kermit’, and ‘mondfe’.(4)
The default for this option depends on the configured target (see section Remote targets supported).
--debug
Turns on the expect
internal debugging output. Debugging output
is displayed as part of the runtest
output, and logged to a file
called ‘dbg.log’. The output shows all attempts at matching the
test output of the tool with the scripted patterns describing expected
output. The output generated with ‘--strace’ also goes into this
file.
--diff filename
Show on the standard output device the differences between the summary
output from the current test run, and a summary from a previous run.
runtest
does not use the Unix utility diff
; instead it
uses an internal Tcl based implementation that is better suited to
output logs (and is reliably portable).
Warning: The ‘--diff’ is not fully implemented in the DejaGnu 1.0 release.
--help
-he
Prints out a short summary of the runtest
options, then exits
(even if you also specify other options).
--mail "name1 name2 …"
--mail name
Send the summary testing output to the specified list, via email. If used with the ‘--diff’ option, only the diff file is mailed. The quotes are only needed if you supply more than one name with a single ‘--mail’ option.
--noreboot
Do not reboot the target board when runtest
initializes.
Usually, when running tests on a separate target board, it is safer to
reboot the target to be certain of its state. However, when developing
test scripts, this option can save a lot of time.
--outdir path
Write output logs in directory path. The default is ‘.’, the
directory where you start runtest
. This option affects only the
summary and the detailed log files ‘tool.sum’ and
‘tool.log’. The DejaGnu debug log ‘dbg.log’ always
appears (when requested) in the local directory.
--runtest "file1 file2 …"
--runtest filename
Specify the names of test files to run. Use this option to choose a subset of the tests available in the ‘--srcdir’ directory. You may supply just filenames with no path; the subdirectory name is not necessary.
--objdir path
Use path as the top directory containing any auxiliary compiled
test code. This defaults to ‘.’. Use this option to locate
pre-compiled test code. You can normally prepare any auxiliary files
needed with make
.
--srcdir path
Use path as the top directory for test scripts to run.
runtest
looks in this directory for any subdirectory whose name
begins with the toolname (specified with ‘--tool’). For instance,
with ‘--tool gdb’, runtest
uses tests in subdirectories
‘gdb.*’ (with the usual shell-like filename expansion). If you do
not use ‘--srcdir’, runtest
looks for test directories under
the current working directory.
--strace n
Turn on internal tracing for expect
, to n levels deep. By
adjusting the level, you can control the extent to which your output
expands multi-level Tcl statements. This allows you to ignore some
levels of case
or if
statements. Each procedure call or
control structure counts as one “level”.
The output is recorded in the same file, ‘dbg.log’, used for output from ‘--debug’.
--target string
string is a full configuration “triple”
name(5) as used by configure
.
Use this option to override the default setting (running native tests).
This option changes the configuration runtest
uses for the
default tool names, and other setup information. See Using configure
in Cygnus configure,
for details about configure
names.
--host string
string is a full configuration “triple” name as used by
configure
. Use this option to override the default setting for
the host. This choice affects only the procedures istarget
,
isnative
, and setup_xfail
.
--name name
name is a name for the particular testing target machine (for
cross testing). If the testing target has IP network support (for
example, RPC
or NFS
), this is the network name for the
target itself. (name is not the configuration string you
specify as a target with configure
; the ‘--name’ option
names a particular target, rather than describing a class of targets.)
For targets that connect in other ways, the meaning of the name
string depends on the connection method. See section Remote targets supported.
--tool tool
tool specifies what set of tests to run, and what initialization module to use. tool is used only for these two purposes: it is not used to name the executable program to test. Executable tool names (and paths) are recorded in ‘site.exp’ (see section Site configuration), and you can override them by specifying Tcl variables on the command line.
For example, including ‘--tool gcc’ on the runtest
command
line runs tests from all test subdirectories whose names match
‘gcc.*’, and uses one of the initialization modules named
‘config/*-gcc.exp’. To specify the name of the compiler (perhaps
as an alternative path to what runtest
would use by default), use
‘GCC=binname’ on the runtest
command line.
--verbose
-v
Turns on more output. Repeating this flag increases the amount of output displayed. Level one (‘-v’) is simply test output. Level two (‘-v -v’) shows messages on options, configuration, and process control. Verbose messages appear in the detailed (‘*.log’) log file, but not in the summary (‘*.sum’) log file.
--version
-V
Prints out the version numbers of DejaGnu, expect
and Tcl.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Once you have the DejaGnu source unpacked and available, the first
requirement before you start using DejaGnu is to configure for the host
(where runtest
executes) and target (where the tools and their
output are tested). Moreover, it is usually best to configure in a
directory separate from the source tree, specifying where to find the
source with the ‘--srcdir’ option to configure
:
configure host-type --target=target-type --srcdir=path
To run tests on a full-fledged operating system, you can specify host and target as the same machine; we call this a native configuration. For native configurations, specifying the target explicitly is not necessary. For example, to configure for a Sun-4 running SunOS:
configure sun4 --srcdir=/usr/cygnus/src/dejagnu
When the host and target are not the same (typically when testing software tools on a “bare board” controlled from a separate general-purpose computer), we refer to a cross configuration. For example, to run tests controlled by Sun-4 on an AMD 29000 board, communicating via the AMD “UDI” protocol:
configure sun4 --target=a29k-amd-udi \ --srcdir=/usr/cygnus/src/dejagnu
You may also want to use the configure
option ‘--prefix’ to
specify where you want DejaGnu and its supporting code installed. By
default, installation is in subdirectories of ‘/usr/local’, but you
can select any alternate directory altdir by including
‘--prefix=altdir’ on the configure
command line.
(This value is captured in the Makefile variables prefix
and exec_prefix
.)
DejaGnu uses Cygnus Configure. See What Configure Does in Cygnus Configure, for more information on its capabilities, conventions, and command-line options.
Aside from the usual effect of preparing and building the executables on
your host, configuring and making DejaGnu creates a file called
‘site.exp’. This file encapsulates information from both
configure
and make
, as default values for DejaGnu tests.
See section Site Configuration.
DejaGnu has some additional requirements for target support, beyond the
general-purpose provisions of Cygnus configure
. runtest
must actively communicate with the target, rather than simply generating
or managing code for the target architecture. Therefore, each tool
requires an initialization module for each target. For new targets, you
must supply a few Tcl procedures to adapt DejaGnu to the target. This
permits DejaGnu itself to remain target independent. See section Initialization module.
Save for a small number of example tests, the DejaGnu distribution
itself does not include any test suites; these are available separately.
Test suites for the GNU compiler (testing both GCC and G++) and for
the GNU binary utilities are distributed in parallel with the
DejaGnu distribution (but packaged as separate files). The test suite
for the GNU debugger is distributed in parallel with each release
of GDB itself, starting with GDB 4.9. After configuring the top-level
DejaGnu directory, unpack and configure the test directories for the
tools you want to test; then, in each test directory, run make
to
build auxiliary programs required by some of the tests.
3.1 Installing DejaGnu | ||
3.2 Site configuration | Site specific configuration file | |
3.3 Initialization module | Platform specific initialization module | |
3.4 Remote targets supported |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
To install DejaGnu in your filesystem (either in ‘/usr/local’, or
as specified by your ‘--prefix’ option to configure
), execute
eg$ make install
‘make install’ does these things for DejaGnu:
runtest
shell script to ‘$exec_prefix/bin’.
Each test suite collection comes with simple installation instructions in a ‘README’ file; in general, the test suites are designed to be unpacked in the source directory for the corresponding tool, and extract into a directory called ‘testsuite’.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The site configuration file, ‘site.exp’, captures
configuration-dependent values from environment variables and propagates
them to the DejaGnu test environment using Tcl variables. This ties the
DejaGnu test scripts into the configure
and make
programs.
DejaGnu supports more than one ‘site.exp’ file. First, make
install
for the DejaGnu distribution places a “master”
‘site.exp’ in each configuration-dependent subdirectory of the
DejaGnu library directory. Second, each directory of tests usually has
another ‘site.exp’, initially created by ‘make check’. The
multiple instances of ‘site.exp’; are loaded in a fixed order built
into DejaGnu (the more local last). You can further override the
default values from this automatic process in a user-editable section of
the file, or by setting variables on the runtest
command line.
runtest
loads
these values first.
runtest
loads these values last, the
individual test configuration can either rely on and use, or override,
any of the global values from the “master” ‘site.exp’.
In either case, you can generate or update ‘site.exp’ by typing
make site.exp
in the appropriate configured directory.
All ‘site.exp’ files have two sections, separated by comment text. The first section starts with:
## these variables are automatically generated by make ## # Do not edit here. If you wish to override these values # add them to the last section |
Do not edit this first section; if you do, your changes are replaced
next time you run make
. The first section ends with this
line:
## All variables above are generated by configure. Do Not Edit ## |
The first section is the part that is generated by make
. It is
essentially a collection of Tcl variable definitions based on
‘Makefile’ environment variables. Since they are generated by
make
, they contain the values as specified by configure
.
(You can also customize these values by using the ‘--site’ option
to configure
.) In particular, this section contains the
‘Makefile’ variables for host and target configuration data.
In the second section, you can override any default values (locally to
DejaGnu) for all the variables. The second section can also contain
your preferred defaults for all the command line options to
runtest
. This allows you to easily customize runtest
for
your preferences in each configured test-suite tree, so that you need
not type options repeatedly on the command line. (The second section
may also be empty, if you do not wish to override any defaults.)
It is usually more convenient to keep these “manual overrides” in the ‘site.exp’ local to each test directory, rather than in the “master” ‘site.exp’ in the DejaGnu library.
To override the values captured in the first part of ‘site.exp’ by
configure
and make
, simply set your own value for any of
these variables in the second part of ‘site.exp’. Set the
variables using the Tcl set
command; for example, to run tests on
an experimental version of GCC, the second part of your
‘site.exp’ might include:
set CC "/working/devo/gcc/xgcc"
Here are the ‘Makefile’ variables set in ‘site.exp’. The default values depend on your configuration. The defaults shown are those built into the DejaGnu distribution, which will appear only if your configuration does not change them. (For example, in cross configurations, program names often reflect the target type.)
CC
The C compiler. The default value is "cc"
.
CFLAGS
The list of options passed to the C compiler. The default value is ‘-g’.
CXX
The C++ compiler. The corresponding variable for make
is
C++
, but Tcl variable names cannot contain the ‘+’
character. The default value is "gcc"
.
CXXFLAGS
The list of options for the C++ compiler. The default value is
"-g"
.
AS
The assembler. The default value is "as"
.
ASFLAGS
The list of options for the assembler. The default value is ""
.
FC
The FORTRAN compiler. The default value is ""
.
FCFLAGS
The list of options for the FORTRAN compiler. The default value is
""
.
LD
The linker. The default value is "ld"
.
LDFLAGS
The list of options for the linker. The default value is ""
.
RANLIB
The program for writing an index to object code archive files.
The default value is "ranlib"
.
RANLIBFLAGS
The options for $RANLIB
. The default value is ""
.
NM
The program that dumps the list of symbols from an object file. The
default value is "nm"
.
NMFLAGS
The options for $NM
. The default value is ""
.
SIZE
This is the program that lists the size of all the sections in the
compiled file. The default value is "size"
.
SIZEFLAGS
The options for $SIZE
. The default value is
""
.
STRIP
The program to strip the symbol table from compiled files
to reduce size. The default value is "strip"
.
STRIPFLAGS
The options for $STRIP
. The default value is ""
.
OBJDUMP
The program that dumps the parts of an object file. The default value is
"objdump"
.
OBJDUMPFLAGS
The options for the object file dumper. The default value is ""
.
LEX
The program that makes lexical analyzers. The default value is
"flex"
.
LEXFLAGS
The options for the lexical analyzer generator. The default value is
""
.
YACC
The “compiler-compiler”, a program to generate parsers. The
default value is "byacc"
.
YACCFLAGS
The options for $YACC
. The default value
is ""
.
AR
The program that builds object code archives. The default value is "ar"
.
ARFLAGS
The options for the archiver. The default value is ""
.
CPP
The C compiler preprocessor. The default value is "gcc -E"
.
CPPFLAGS
The options for the C preprocessor. The default value is ""
.
GDB
The C and C++ debugger. The default value is "gdb"
.
GDBFLAGS
The options for the debugger. The default value is ""
.
DIFF
The program that finds differences in text files. The default
value is "diff"
.
DIFFFLAGS
The options for the diff
program. The default value is ""
.
The following variables, also in ‘site.exp’, extend the standard
set of configuration variables. You should never redefine these; they
permit runtest
to determine its configuration at runtime.
host_os
The host operating system name, as seen by ‘config.sub’. The default depends on the value you specify with ‘configure --host’ for your configuration.
host_alias
A shorter name for the host system. The default depends on the value you specify with ‘configure --host’ for your configuration.
host_cpu
The host processor type. The default depends on the value you specify with ‘configure --host’ for your configuration.
host_vendor
The manufacturer of the host system. The default depends on the value you specify with ‘configure --host’ for your configuration.
host_triplet
This is the canonical name produced by configure
.
Its value is the same as ‘$host_cpu-$host_vendor-$host_os’.
target_os
The target operating system name, as seen by ‘config.sub’. The default depends on the value you specify with ‘configure --target’.
target_alias
A possibly shorter name for the target system. The default depends on the value you specify with ‘configure --target’.
target_cpu
The target processor type. The default depends on the value you specify with ‘configure --target’.
target_vendor
The manufacturer of the host system. The default depends on the value you specify with ‘configure --target’.
target_triplet
This is the canonical name produced by configure
. Its valeu is
the same as ‘$target_cpu-$target_vendor-$target_os’.
In the user editable second section of ‘site.exp’, you can not only
override the configuration variables captured in the first section, but
also specify default values for all the runtest
command line
options. Save for ‘--debug’, ‘--help’, and ‘--version’,
each command line option has an associated Tcl variable. Use the Tcl
set
command to specify a new default value (as for the
configuration variables). The following table describes the
correspondence between command line options and variables you can set in
‘site.exp’. See section Running the Tests, for
explanations of the command-line options.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The initialization module (or “init file”) has two purposes: to provide tool and target dependent procedures, and to start up an interactive tool to the point where it is ready to operate. The latter includes establishing communications with the target. All the tests for interactive programs assume that the tool is already running and communicating. Initialization modules for non-interactive programs may only need to supply the support functions.
Each test suite directory must contain (in its ‘config’
subdirectory) a separate initialization module for each target. The
appropriate init file is identified by name: it must be named
‘$target_os.exp’, where target_os
is the Tcl variable
recording the configuration name for the target operating system.
If the test suite contains tests for more than one tool, it can either
use a single init file for all the tools, or segregate the init files by
tool using a more elaborate naming convention:
‘$target_os-$tool.exp’, where tool
is the Tcl variable
identifying tests for a particular tool. It is better to avoid this
convention if possible, since it may lead to unacceptable constraints on
tool names due to systems with low limits on file-name length. You can
always segregate tests for different tools into different test-suite
directories, instead.
In a test running native, writing the Tcl code for initialization is usually quite simple. For cross configurations, however, more elaborate instructions are usually needed to describe how to talk to a remote target.
There are conventionally four procedures in each initialization module.
Their names begin with ‘$tool’, the string that identifies tests
for a particular tool: $tool_start
, $tool_load
,
$tool_exit
, and $tool_version
. For example, the start
procedure for GDB is called gdb_start
. (Since start
procedures are used differently for batch and interactive tools,
however, runtest
itself never calls the start procedure. Init
files for interactive tools are expected to end by running the start
procedure.)
The initialization module is also a good place to call load_lib
to get any collections of utility procedures meant for a family of test
cases, and to set up default values for any additional Tcl variables
needed for a specific set of tests.
See section Target dependent functions, for full descriptions of these procedures.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
You can set the target name and the connect mode in the ‘site.exp’
file (using the Tcl variables ‘targetname’ and ‘connectmode’,
respectively), or on the runtest
command line (using
‘--target’ and ‘--connect’).
The DejaGnu distribution includes support for the following remote
targets. If you suspect a communication problem, try running the
connection interactively from expect
. (There are three ways of
running expect
as an interactive interpreter. You can run
expect
with no arguments, and control it completely
interactively; or you can use ‘expect -i’ together with other
command-line options and arguments; or you can run the command
interpreter
from any expect
procedure. Use return
to get back to the calling procedure (if any), or return -tcl
to
make the calling procedure itself return to its caller; use exit
or end-of-file to leave expect
altogether.) Run the program
whose name is recorded in ‘$connectmode’, with the arguments in
‘$targetname’, to establish a connection. You should at least be
able to get a prompt from any target that is physically connected.
See section Adding a target, for some discussion of writing
and debugging new targets.
Configure DejaGnu for target ‘a29k-amd-udi’. (Cygnus
configure
also recognizes the abbreviation ‘udi29k’.) Then,
to run tests, use the runtest
target name to specify whether you
want to use a simulator, or a particular hardware board. The particular
string to use with ‘--target’ will depend on your UDI setup file,
‘udi_soc’ (if ‘udi_soc’ is not in your working directory, the
environment variable ‘UDICONF’ should contain a path to this file).
For example, if your UDI setup file includes these lines:
iss AF_UNIX * isstip -r /home/gnu/29k/src/osboot/sim/osboot mon AF_UNIX * montip -t serial -baud 9600 -com /dev/ttyb |
use ‘--target iss’ to run tests on the simulator, and ‘--target mon’ to run tests on the 29K hardware. See the manufacturer’s manuals for more information on UDI and ‘udi_soc’.
The default connect protocol is ‘mondfe’ with either back end.
mondfe
is the only shell DejaGnu supports for UDI targets.
Warning: This target requires GDB version 4.7.2 (or
greater). Earlier versions of GDB do not fully support the
load
command on this target, so DejaGnu has no way to load
executable files from the debugger.
Configure DejaGnu for any remote target matching ‘m68k-*’.
Warning: Most ‘m68k-*’ configurations run all tests only for native testing (when the target is the same as the host). When you specify most of these targets for a cross configuration, you will only be able to use tests that run completely within the host (for example, tests of the binary utilities such as the archiver; or compiler tests that only generate code rather than running it).
To run a.out or COFF binaries on a remote M68K, you must configure DejaGnu for a particular target board. ‘m68k-abug’ is an example. (In general for an embedded environment, because it dosen’t have absolute addresses, a.out is not a good choice for output format in any case; most often S-records or Hex-32 are used instead.)
Configure for ‘m68k-abug-aout’ or ‘m68k-abug-coff’ (as a
target). This boot monitor can only download S-records; therefore, the
DejaGnu tests for this environment require a linker command script to
convert either output format to S-records, setting the default addresses
for .text
, .bss
, and .data
.
With this configuration, the default for ‘--connect’ is ‘tip’.
‘tip’ is the only communications protocol supported for connecting
to ‘m68k-abug-*’ targets. ‘tip’ uses an ASCII downloader
(the ~put
command) to load S-records into the target board. The
‘--target’ string must be a machine name that tip
understands (for example, on some tip
implementations it must be
an entry from the initialization file for tip
; this file is
sometimes called ‘/etc/remote’).
See your system documentation for information on how to create new entries in ‘/etc/remote’. (Some UNIX systems are distributed with at least one default entry with a name resembling ‘hardwire’; if your system has one, you can edit it, or make a modified copy with a new name.) When you have a working ‘/etc/remote’ entry abugtarget, you should be able to type ‘tip abugtarget’, and get the prompt ‘135ABUG>’ from the board. Use the same abugtarget string with ‘runtest --target’.
This is the same in functionality as the MVME board running the
BUG
boot monitor. Only the monitor commands and the addresses are
different.
Configure DejaGnu for either ‘m68k-wrs-vxworks’ (abbreviated ‘vxworks68’) or ‘i960-wrs-vxworks’ (abbreviated ‘vxworks960’). Since both targets support IP addressing, specify the network address (for example, a host name from ‘/etc/hosts’) with ‘--target’.
The default connect protocol is ‘rlogin’, but you can use any of ‘--connect rlogin’, ‘--connect telnet’, or ‘--connect rsh’.
Test scripts need no special code to load programs into these targets; since VxWorks supports NFS, all you must do is ensure test programs are on an exported filesystem.
When you compile for VxWorks, use the linker ‘-r’ option to make the linker output relocatable—at least if you want to use library routines. Many standard C routines are included in VxWorks; often no additional libraries are needed. See your VxWorks system documentation for additional details.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
DejaGnu is entirely written in expect
, which uses Tcl as a
command language. expect
serves as a very programmable shell;
you can run any program, as with the usual Unix command shells—but
once the program is started, your expect
script has fully
programmable control of its input and output. This does not just apply
to the programs under test; expect
can also run any auxiliary
program, such as diff
or sh
, with full control over its
input and output.
The most important part of DejaGnu—and the largest body of code—is the set of test suites distributed separately for each GNU tool. Future releases of GNU tools will include even more tests, developed throughout the free software community.
runtest
, of course, is the glue to tie together and manage the
test scripts. The runtest
program is actually a simple Bourne
shell script that locates a copy of the expect shell and then starts the
main Tcl code, runtest.exp
. runtest.exp
itself has these
essential functions:
runtest.exp
locates the tests
by exploiting a straightforward naming convention based on the string
you specify with the ‘--tool’ option.
expect
.
4.1 Conventions for using tool names | ||
4.2 DejaGnu procedures | DejaGnu provides these Tcl procedures | |
4.3 Target dependent functions | Procedures supplied by the init module | |
4.4 The files DejaGnu reads | The files DejaGnu depends on | |
4.5 The files DejaGnu writes | The files DejaGnu produces |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
DejaGnu uses ‘$tool’, the name of the tool under test, to tie together the testing configuration in a straightforward but flexible way.
Each initialization module is tool-dependent; their names optionally use the tool name as a suffix. If the ‘testsuite/config’ directory is only for one tool, then the tool name need not be part of the init file name. See section Initialization module.
‘$tool’ is not used to invoke the tool, since sites that run
multiple configurations of a particular tool often call each
configuration by a different name. runtest
uses the
configuration-dependent variables captured in ‘site.exp’ to
determine how to call each tool.
runtest
also uses tool names to find directories containing
tests. runtest
scans the source directory (specified with
--srcdir
) for all directories whose names start with the tool
name followed by a period. For instance, directories that start with
‘g++.’ contain G++ tests. To add a new test, just put it in
any directory (including, if you wish, an entirely new directory) whose
name follows this convention.
A test is any file in an appropriately named subdirectory whose name
ends in ‘.exp’ (the conventional way of naming expect
scripts). These simple naming conventions make it as simple as possible
to install new tests: all you must do is put the test in the right
directory.
runtest
sorts the tests in each subdirectory by name (using the
Tcl lsort
command) and runs them in the resulting order.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
DejaGnu provides these Tcl procedures for use in test scripts.
You can also use any standard expect
or Tcl function.
error "string"
Declares detection of a severe error in the test case itself.
error
writes in the log files a message beginning with
‘ERROR’, appending the argument string. (This is a Tcl
built-in; for more detail, @pxref{Tcl Commands, Built-in commands}).
fail "string"
Declares a test to have failed. fail
writes in the
log files a message beginning with ‘FAIL’ (or XFAIL
, if
failure was expected), appending the argument string.
ishost "host"
Tests for a particular host environment. If the currently
configured host matches the argument string, the result is 1
;
otherwise the result is 0
. host must be a full three-part
configure
host name; in particular, you may not use the shorter
nicknames supported by configure
(but you can use wildcard
characters to specify sets of names).
istarget "target"
Tests for a particular target environment. If the currently
configured target matches the argument string, the result is 1
;
otherwise the result is 0
. target must be a full
three-part configure
target name; in particular, you may not use
the shorter nicknames supported by configure
(but you can use
wildcard characters to specify sets of names).
isnative
Tests whether the current configuration has the same host and target.
When it runs in a native configuration this procedure returns a
1
; otherwise it returns a 0
.
load_lib "library-file"
Loads the file library-file by searching a fixed path built into
runtest
. If DejaGnu has been installed, it looks in a path
starting with the installed library directory. If DejaGnu is not
installed, it defaults to the current directory. In either case, it
then looks in the current directory for a directory called lib
.
If there are duplicate definitions, the last one loaded takes precedence
over the earlier ones.
pass "string"
Declares a test to have passed. pass
writes in the
log files a message beginning with ‘PASS’ (or XPASS
, if
failure was expected), appending the argument string.
setup_xfail "config [bugno]"
Declares that the test is expected to fail on a particular set of
configurations. The config argument must be a list of full
three-part configure
target name; in particular, you may not use
the shorter nicknames supported by configure
(but you can use the
common shell wildcard characters to specify sets of names). The
bugno argument is optional, and used only in the logging file
output; use it as a link to a bug-tracking system such as GNATS
(see Overview in Tracking Bugs With GNATS).
Warning: you must call either pass
or fail
after
using setup_xfail
in a test case. Otherwise, the
expected-failure declaration applies to whatever test runs next,
leading to surprising results.
verbose "string" number
Test cases can use this function to issue helpful messages depending on
the number of ‘--verbose’ options on the runtest
command
line. It prints string if the value of the variable
verbose
is higher than the optional number. The default
value for number is 1.
warning "string"
Declares detection of a minor error in the test case itself.
warning
writes in the log files a message beginning with
‘WARNING’, appending the argument string. Use warning
rather than error
for cases (such as communication failure
to be followed by a retry) where the test case can recover from the
error.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Each combination of target and tool requires some target-dependent procedures. The names of these procedures have a common form: the tool name, followed by an underbar ‘_’, and finally a suffix describing the procedure’s purpose. For example, a procedure to extract the version from GDB is called ‘gdb_version’. See section Initialization Module, for a discussion of how DejaGnu arranges to find the right procedures for each target.
tool_start
Starts a particular tool. For an interactive tool,
tool_start
starts and initializes the tool, leaving the
tool up and running for the test cases; an example is gdb_start
,
the start function for GDB. For a batch oriented tool,
tool_start
is optional; the recommended convention is to
let tool_start
run the tool, leaving the output in a
variable called comp_output
. Test scripts can then analyze
‘$comp_output’ to determine the test results. An example of this
second kind of start function is gcc_start
, the start function
for GCC.
runtest
itself does not call tool_start
. The
initialization module ‘tool_init.exp’ must call
tool_start
for interactive tools; for batch-oriented tools,
each individual test script calls tool_start
(or makes
other arrangements to run the tool).
tool_load
Loads something into a tool. For an interactive tool, this conditions
the tool for a particular test case; for example, gdb_load
loads
a new executable file into the debugger. For batch oriented tools,
tool_load
may do nothing—though, for example, the
GCC support uses gcc_load
to load and run a binary on the
target environment. Conventionally, tool_load
leaves the
output of any program it runs in a variable called ‘exec_output’.
Writing tool_load
can be the most complex part of extending
DejaGnu to a new tool or a new target, if it requires much communication
coding or file downloading.
Test scripts call tool_load
.
tool_exit
Cleans up (if necessary) before runtest
exits. For interactive
tools, this usually ends the interactive session. You can also use
tool_exit
to remove any temporary files left over from the
tests.
runtest
calls tool_exit
.
tool_version
Prints the version label and number for tool. This is called by the DejaGnu procedure that prints the final summary report. The output should consist of the full path name used for the tested tool, and its version number.
runtest
calls tool_version
.
The usual convention for return codes from any of these procedures
(although it is not required by runtest
) is to return 0
if
the procedure succeeded, 1
if it failed, and -1
if there
was a communication error.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
There are a few files used by DejaGnu for input. The most important one
is ‘site.exp’, which contains the configuration data.
‘runtest.exp’ implements the main test framework; the
runtest
program used to invoke DejaGnu is a short shell script
generated by make
during the configuration process.
The only other general-purpose input files are collections of
expect
utilities kept in the ‘lib’ subdirectory. These
files are not used by all tests, but are used (via the Tcl command
source
) often enough to warrant a common definition.
‘lib/remote.exp’ defines these functions, for establishing and managing communications:
Procedures to establish a connection: Each of these procedures
tries to establish the connection up to three times before returning.
Warnings (if retries will continue) or errors (if the attempt is
abandoned) report on communication failures. The result for any of
these procedures is either -1
, when the connection cannot be
established, or the spawn ID returned by the expect
command
spawn
.
telnet hostname
rlogin hostname
rsh hostname
IP network procedures. hostname refers to the IP address or name (for example, an entry in ‘/etc/hosts’) for this target. The procedure names reflect the Unix utility used to establish a connection.
tip port
Serial line procedure. Connect using the Unix utility tip
.
port must be a name from the tip
configuration file
‘/etc/remote’. Often, this is called ‘hardwire’, or something
like ‘hardwire-a’. This file holds all the configuration data for
the serial port.
kermit port bps
Serial line procedure. Connect using the program kermit
.
port is the device name, e.g. ‘/dev/ttyb’. bps is
the line speed to use (in bits per second) for the connection.
Procedures to manage a connection:
tip_download spawnid file
Download ‘file’ to the process spawnid (the value
returned when the connection was established), using the ~put
command under tip
. Most often used for single board computers
that require downloading programs in ASCII S-records. Returns
1
if an error occurs, 0
otherwise.
exit_remote_shell spawnid
Exits a remote process started by any of the connection procedures. spawnid is the result of the connection procedure that started the remote process.
‘lib/utils.exp’ defines these utility procedures:
getdirs path
getdirs path pattern
Returns a list of all the directories in path that match
pattern. If you do not specify pattern, getdirs
assumes ‘*’. You may use the common shell wildcard characters in
pattern.
find path pattern
Search for files whose names match pattern (using shell wildcard characters for filename expansion). Search subdirectories recursively, starting at path. The result is the list of files whose names match; if no files match, the result is empty. Filenames in the result include all intervening subdirectory names.
which binary
Searches the execution path for an executable file binary, like
the the BSD which
utility. This procedure uses the shell
environment variable ‘PATH’. It returns 0
if the binary is
not in the path, or if there is no ‘PATH’ environment variable. If
binary is in the path, it returns the full path to binary.
grep filename regexp
grep filename regexp line
Search the file called filename (a fully specified path) for lines
that contain a match for regular expression regexp. The result is
a list of all the lines that match. If no lines match, the result is an
empty string. Specify regexp using the standard regular
expression style used by the Unix utility program grep
.
Use the optional third argument ‘line’ to start lines in the result with the line number in filename.
prune list pattern
Remove elements of the Tcl list list. Elements are fields delimited by spaces. The result is a copy of list, without any elements that match pattern. You can use the common shell wildcard characters to specify pattern.
Most tools also have a library file associated with the tool, containing procedures found useful by other test developers. These are especially helpful when the tests span several subdirectories.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
runtest
always writes two kinds of output files: summary logs and
detailed logs. The contents of both of these are determined by your
tests.
For troubleshooting, a third kind of output file is useful: use
‘--debug’ to request an output file showing details of what
expect
is doing internally.
4.5.1 Summary log | Files that summarize tests | |
4.5.2 Detailed log | Files that contain complete test results | |
4.5.3 Logging expect internal actions | Logging expect internal actions |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
runtest
always produces a summary output file
‘tool.sum’. This summary shows the names of all test files
run; for each test file, one line of output from each pass
command (showing status ‘PASS’ or ‘XPASS’) or fail
command (status ‘FAIL’ or ‘XFAIL’) (or from error
or
warning
commands); trailing summary statistics that count passing
and failing tests (expected and unexpected); and the full pathname and
version number of the tool tested. (All four possible outcomes, and all
errors, are always reflected in the summary output file, regardless of
whether or not you specify ‘--all’.)
For example, after ‘runtest --tool binutils’, look for a summary
log in ‘binutils.sum’. Normally, runtest
writes this file
in your current working directory; use the ‘--output’ option to
select a different directory.
Here is a short sample summary log:
Test Run By rob on Mon May 25 21:40:57 PDT 1992 === gdb tests === Running ./gdb.t00/echo.exp ... PASS: Echo test Running ./gdb.all/help.exp ... PASS: help add-symbol-file PASS: help aliases PASS: help breakpoint "bre" abbreviation FAIL: help run "r" abbreviation Running ./gdb.t10/crossload.exp ... PASS: m68k-elf (elf-big) explicit format; loaded XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types === gdb Summary === # of expected passes 5 # of expected failures 1 # of unexpected successes 0 # of unexpected failures 1 /usr/latest/bin/gdb version 4.6.5 -q |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
runtest
also saves a detailed log file ‘tool.log’,
showing any output generated by tests as well as the summary output.
For example, after ‘runtest --tool binutils’, look for a detailed
log in ‘binutils.log’. Normally, runtest
writes this file
in your current working directory; use the ‘--output’ option to
select a different directory.
Here is a brief example showing a detailed log for G++ tests:
Test Run By rob on Mon May 25 21:40:43 PDT 1992 === g++ tests === --- Running ./g++.other/t01-1.exp --- PASS: operate delete --- Running ./g++.other/t01-2.exp --- FAIL: i960 bug EOF p0000646.C: In function `int warn_return_1 ()': p0000646.C:109: warning: control reaches end of non-void function p0000646.C: In function `int warn_return_arg (int)': p0000646.C:117: warning: control reaches end of non-void function p0000646.C: In function `int warn_return_sum (int, int)': p0000646.C:125: warning: control reaches end of non-void function p0000646.C: In function `struct foo warn_return_foo ()': p0000646.C:132: warning: control reaches end of non-void function --- Running ./g++.other/t01-4.exp --- FAIL: abort 900403_04.C:8: zero width for bit-field `foo' --- Running ./g++.other/t01-3.exp --- FAIL: segment violation 900519_12.C:9: parse error before `;' 900519_12.C:12: Segmentation violation /usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal === g++ Summary === # of expected passes 1 # of expected failures 3 # of unexpected successes 0 # of unexpected failures 0 /usr/ps/bin/g++ version cygnus-2.0.1 |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
expect
internal actionsWith the ‘--debug’ option, you can request a log file showing the
output from expect
itself, running in debugging mode. This file
(‘dbg.log’, in the directory where you start runtest
) shows
each pattern expect
considers in analyzing test output.
This file reflects each send
command, showing the string sent as
input to the tool under test; and each expect
command, showing
each pattern it compares with the tool output.
The log messages for expect
begin with a message of the form
expect: does {tool output} (spawn_id n) match pattern {expected pattern}?
For every unsuccessful match, expect
issues a ‘no’ after
this message; if other patterns are specified for the same
expect
command, they are reflected also, but without the first
part of the message (‘expect…match pattern’).
When expect
finds a match, the log for the successful match ends
with ‘yes’, followed by a record of the expect
variables set
to describe a successful match. Here is an excerpt from the debugging
log for a GDB test:
send: sent {break gdbme.c:34\n} to spawn id 6 expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? no {.*\(gdb\) $}? no expect: does {} (spawn_id 0) match pattern {<return>}? no {\(y or n\) }? no {buffer_full}? no {virtual}? no {memory}? no {exhausted}? no {Undefined}? no {command}? no break gdbme.c:34 Breakpoint 8 at 0x23d8: file gdbme.c, line 34. (gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes expect: set expect_out(0,start) {18} expect: set expect_out(0,end) {71} expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } expect: set expect_out(spawn_id) {6} expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } PASS: 70 0 breakpoint line number in file |
This example exhibits three properties of expect
and DejaGnu that
might be surprising at first glance:
expect
begins attempting to match the patterns supplied
immediately; often, the first pass is against incomplete output (or
completely before all output, as in this case).
runtest
first tries the patterns specified explicitly in each
test case; however, if none of these patterns matches, runtest
tries a default list of patterns designed to try to catch common errors
or oversights in writing test scripts.
The default patterns (like the debugging log itself) are primarily
useful while developing test scripts. The actions for these default
patterns produce messages starting with ‘ERROR’ on the
runtest
standard output, and in the detailed log file. Here is a
list of these default patterns:
‘{<return>}’
Checks for an unexpected <RET> prompt. This may happen, for example, if the tested tool can filter output through a pager.
‘{\(y or n\) }’
A common pattern in prompts requiring user input. This will trap some unexpected prompts from a program.
‘{buffer_full}’
Detects overflow of the internal expect
buffer for output from
the tested tool. The default buffer size is 2000 characters. You can
specify a larger buffer, if some of your tests need one, with the
match_max
Tcl function provided by expect
.
‘{virtual} {memory} {exhausted}’
The message ‘virtual memory exhausted’ from GDB (and perhaps other tested tools) may indicate a problem with the test script itself. Sometimes a pattern with poor closure (see section Hints on Writing a Test Case) fills up available memory, leaving very little for the process that was spawned.
‘{Undefined} {command}’
The message ‘Undefined command’ from GDB (and perhaps other tested tools) may be a response to a test script that sends a nonexistent command. Sometimes commands disappear as updates are released; this may also arise from a simple typing mistake in a test script.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
5.1 Writing a test case | ||
5.2 Debugging a test case | ||
5.3 Adding a test case to DejaGnu | ||
5.4 Hints on writing a test case | ||
5.5 Special variables used by test cases |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The easiest way to prepare a new test case is to base it on an existing one for a similar situation. There are two major categories of tests: batch or interactive. Batch oriented tests are usually easier to write.
The GCC tests are a good example of batch oriented tests. All
GCC tests consist primarily of a call to a single common procedure,
since all the tests either have no output, or only have a few warning
messages when successfully compiled. Any non-warning output is a test
failure. All the C code needed is kept in the test directory. The test
driver, written in expect
, need only get a listing of all the C
files in the directory, and compile them all using a generic procedure.
This procedure and a few others supporting for these tests are kept in
the library module ‘lib/c-torture.exp’. Most tests of this kind
use very few expect
features, and are coded almost purely in Tcl.
Writing the complete suite of C tests, then, consisted of these steps:
expect
library code.
glob
for filename expansion with
wildcards) and call a Tcl procedure with each filename. It also checks
for a few errors from the testing procedure.
Testing interactive programs is intrinsically more complex. Tests for most interactive programs require some trial and error before they are complete.
However, some interactive programs can be tested in a simple fashion reminiscent of batch tests. For example, the GDB distribution itself includes a testing procedure that covers most of the DejaGnu GDB tests. This procedure is very robust, and has already undergone much more debugging and error checking than many other DejaGnu test cases. Accordingly, the best approach was simply to encapsulate the existing GDB tests, for reporting purposes, in DejaGnu tests that use common library code.
‘gdb.t10/crossload.exp’ is a good example of an interactive test.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
These are the kinds of debugging information available from DejaGnu:
detail
directory. To do the same for new tests,
use the ‘verbose’ variable to control how much output to generate;
this will make it easier for other people running the test to debug it
if necessary. Whenever possible, if ‘$verbose’ is 0
, there
should be no output other than the output from pass
, fail
,
error
, and warning
. Then, to whatever extent is
appropriate for the particular test, allow successively higher values of
‘verbose’ to generate more information. Be kind to other
programmers who use your tests: provide for a lot of debugging
information.
expect
.
There is a command line options for each; both forms of debugging output
are recorded in the file dbg.log
in the current directory.
Use ‘--debug’ for information from the expect
level; it
generates displays of the expect
attempts to match the tool
output with the patterns specified (see section Debug Log). This
output can be very helpful while developing test scripts, since it shows
precisely the characters received. Iterating between the latest attempt
at a new test script and the corresponding ‘dbg.log’ can allow you
to create the final patterns by “cut and paste”. This is sometimes
the best way to write a test case.
Use ‘--strace’ to see more detail at the Tcl level; this shows how Tcl procedure definitions expand, as they execute. The associated number controls the depth of definitions expanded; see the discussion of ‘--strace’ in Running the Tests.
runtest
turns on the expect
command log_user
. This command prints
all expect
actions to the expect
standard output, to the
detailed log file, and (if ‘--debug’ is on) to ‘dbg.log’.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
There are two slightly different ways to add a test case. One is to add the test case to an existing directory. The other is to create a new directory to hold your test. The existing test directories represent several styles of testing, all of which are slightly different; examine the directories for the tool of interest to see which (if any) is most suitable.
Adding a GCC test can be very simple: just add the C code to any directory beginning with ‘gcc.’ and it runs on the next ‘runtest --tool gcc’.
To add a test to GDB, first add any source code you will need to
the test directory. Then you can either create a new expect
file,
or add your test to an existing one (any file with a ‘.exp’
suffix). Creating a new ‘.exp’ file is probably a better idea if
the test is significantly different from existing tests. Adding it as a
separate file also makes upgrading easier. If the C code has to be
already compiled before the test will run, then you’ll have to add it to
the ‘Makefile.in’ file for that test directory, then run
configure
and make
.
Adding a test by creating a new directory is very similar:
make
and configure
next run, they include the new directory.
Makefile.in
and a configure.in
. See What Configure Does in configure.info.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
There may be useful existing procedures already written for your test in the ‘lib’ directory of the DejaGnu distribution. See section The files DejaGnu reads.
It is safest to write patterns that match all the output
generated by the tested program; this is called closure. If a
pattern does not match the entire output, any output that remains will
be examined by the next expect
command. In this
situation, the precise boundary that determines which expect
command sees what is very sensitive to timing between the expect
task and the task running the tested tool; as a result, the test may
sometimes appear to work, but is likely to have unpredictable results.
The best way to ensure closure is to use the ‘-re’ option for the
expect
command to write the pattern as a full regular
expressions; then you can match the end of output using a ‘$’. It
is also a good idea to write patterns that match all available output by
using ‘.*\’ after the text of interest; this will also match any
intervening blank lines. Sometimes an alternative is to match end of
line using ‘\r’ or ‘\n’, but this is usually too dependent on
terminal settings.
Always escape punctuation, such as ‘(’ or ‘"’, in your patterns; for example, write ‘\(’. If you forget to escape punctuation, you will usually see an error message like ‘extra characters after close-quote’.
If you have trouble understanding why a pattern does not match the
program output, try using the ‘--debug’ option to runtest
,
and examine the debug log carefully. See section Debug Log.
Be careful not to neglect output generated by setup rather than by the
interesting parts of a test case. For example, while testing GDB,
I issue a send ‘set height 0\n’ command. The purpose is simply to
make sure GDB never calls a paging program. The ‘set height’
command in GDB does not generate any output; but running any
command makes GDB issue a new ‘(gdb) ’ prompt. If there were
no expect
command to match this prompt, the output ‘(gdb) ’
begins the text seen by the next expect
command—which might
make that pattern fail to match.
To preserve basic sanity, I also recommended that no test ever pass if
there was any kind of problem in the test case. To take an extreme
case, tests that pass even when the tool will not spawn are misleading.
Ideally, a test in this sort of situation should not fail either.
Instead, print an error message by calling one of the DejaGnu procedures
error
or warning
.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Your test cases can use these variables, with conventional meanings (as well as the variables saved in ‘site.exp’ see section Site configuration):
These variables are available to all test cases.
prms_id
The tracking system (e.g. GNATS) number identifying a corresponding bugreport. (‘0’ if you do not specify it in the test script.)
bug_id
An optional bug id; may reflect a bug identification from another organization. (‘0’ if you do not specify it.)
subdir
The subdirectory for the current test case.
These variables should never be changed. They appear in most tests.
expect_out(buffer)
The output from the last command. This is an internal variable set by
expect
.
exec_output
This is the output from a tool_load
command. This only
applies to tools like GCC and GAS which produce an object
file that must in turn be executed to complete a test.
comp_output
This is the output from a tool_start
command. This is
conventionally used for batch oriented programs, like GCC and
GAS, that may produce interesting output (warnings, errors) without
further interaction.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The most common ways to extend the DejaGnu framework are: adding a suite
of tests for a new tool to be tested; adding support for testing on a
new target; and porting runtest
to a new host.
6.1 Writing Tests for a New Tool | How to add tests for a new tool | |
6.2 Adding a target | How to add a new target | |
6.3 Porting to a new host | Porting DejaGnu to a new host |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
In general, the best way to learn how to write (code or even prose) is
to read something similar. This principle applies to test cases and to
test suites. Unfortunately, well-established test suites have a way of
developing their own conventions: as test writers become more
experienced with DejaGnu and with Tcl, they accumulate more utilities,
and take advantage of more and more features of expect
and Tcl in
general.
Inspecting such established test suites may make the prospect of creating an entirely new test suite appear overwhelming. Nevertheless, it is quite straightforward to get a new test suite going.
There is one test suite that is guaranteed not to grow more elaborate
over time: both it and the tool it tests were created expressly to
illustrate what it takes to get started with DejaGnu. The
‘example/’ directory of the DejaGnu distribution contains both an
interactive tool called calc
, and a test suite for it. Reading
this test suite, and experimenting with it, is a good way to supplement
the information in this section. (Thanks to Robert Lupton for creating
calc
and its test suite—and also the first version of this
section of the manual!)
To help orient you further in this task, here is an outline of the steps to begin building a test suite for a program example.
testsuite
):
eg$ cd testsuite/
target_abbrev
; this value is the link to the init file you will
write soon. (For simplicity, we assume the target environment is Unix,
and use ‘unix’ as the value.)
What else is needed in ‘configure.in’ depends on the requirements
of your tool, your intended test environments, and which
configure
system you use. This example is a minimal
configure.in
for use with Cygnus Configure. (For an alternative
based on the FSF autoconf
system, see the calc
example
distributed with DejaGnu.) Replace example with the name of your
program:
# This file is a shell script fragment # for use with Cygnus configure. srctrigger="example.0" srcname="The DejaGnu example tests" # per-host: # per-target: # everything defaults to unix for a target target_abbrev=unix # post-target: |
configure
to
build your ‘Makefile’. Its leading section should as usual contain
the values that configure
may override:
srcdir = . prefix = /usr/local exec_prefix = $(prefix) bindir = $(exec_prefix)/bin libdir = $(exec_prefix)/lib tooldir = $(libdir)/$(target_alias) datadir = $(exec_prefix)/lib/deja-gnu RUNTEST = runtest RUNTESTFLAGS = FLAGS_TO_PASS = #### host, target, and site specific Makefile frags come in here. |
This should be followed by the standard targets at your site. To begin with, they need not do anything—for example, these definitions will do:
all: info: install-info: install: uninstall: clean: -rm -f *~ core *.info* |
It is also a good idea to make sure your ‘Makefile’ can rebuild itself if ‘Makefile.in’ changes, with a target like this (which works for either Cygnus or FSF Configure):
Makefile : $(srcdir)/Makefile.in $(host_makefile_frag) \ $(target_makefile_frag) $(SHELL) ./config.status |
You also need to include two targets important to DejaGnu: check
,
to run the tests, and site.exp
, to set up the Tcl copies of
configuration-dependent values. The check
target must run
‘runtest --tool example’:
check: site.exp all $(RUNTEST) $(RUNTESTFLAGS) $(FLAGS_TO_PASS) \ --tool example --srcdir $(srcdir) \ --target $(target_canonical) |
The site.exp
target should usually set up (among other things!) a
Tcl variable for the name of your program:
site.exp: ./config.status Makefile @echo "Making a new config file..." -@rm -f ./tmp? @touch site.exp -@mv site.exp site.bak @echo "## these variables are automatically\ generated by make ##" > ./tmp0 @echo "# Do not edit here. If you wish to\ override these values" >> ./tmp0 @echo "# add them to the last section" >> ./tmp0 @echo "set host_os ${host_os}" >> ./tmp0 @echo "set host_alias ${host_alias}" >> ./tmp0 @echo "set host_cpu ${host_cpu}" >> ./tmp0 @echo "set host_vendor ${host_vendor}" >> ./tmp0 @echo "set target_os ${target_os}" >> ./tmp0 @echo "set target_alias ${target_alias}" >> ./tmp0 @echo "set target_cpu ${target_cpu}" >> ./tmp0 @echo "set target_vendor ${target_vendor}" >> ./tmp0 @echo "set host_triplet ${host_canonical}" >> ./tmp0 @echo "set target_triplet ${target_canonical}" >> ./tmp0 @echo "set tool binutils" >> ./tmp0 @echo "set srcdir ${srcdir}" >> ./tmp0 @echo "set objdir `pwd`" >> ./tmp0 @echo "set examplename example" >> ./tmp0 @echo "## All variables above are generated by\ configure. Do Not Edit ##" >> ./tmp0 @cat ./tmp0 > site.exp @cat site.bak | sed \ -e '1,/^## All variables above are.*##/ d' \ >> site.exp -@rm -f ./tmp? |
eg$ mkdir config
target_abbrev
value, so call it ‘config/unix.exp’.
This is the file that contains the target-dependent procedures;
fortunately, most of them do not have to do very much in order for
runtest
to run.
If example is not interactive, you can get away with this minimal ‘unix.exp’ to begin with:
proc foo_exit {} {} proc foo_version {} {} |
If example is interactive, however, you might as well define a start routine and invoke it by using an init file like this:
proc foo_exit {} {} proc foo_version {} {} proc foo_start {} { global examplename spawn $examplename expect { -re "" {} } } foo_start |
eg$ mkdir example.0
send_user "Testing: one, two...\n" |
eg$ configure
(You may have to specify more of a path, if a suitable configure
is not available in your execution path.)
Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 === example tests === Running ./example.0/first-try.exp ... Testing: one, two... === example Summary === # of expected passes 0 # of expected failures 0 # of unexpected successes 0 # of unexpected failures 0 |
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Supporting a new target requires one thing: an initialization module for each tool you wish to test there. This module contains all the operating system dependent code. See section Initialization Module, for a discussion of naming conventions that will enable DejaGnu to use your new init files.
Most of the work is in getting the communications right. Communications code (for several situations involving IP networks or serial lines) is available in a DejaGnu library file, ‘lib/remote.exp’. See section The files DejaGnu reads.
Usually the best way to write a new initialization module is to edit an existing initialization module; some trial and error will be required. If necessary, remember you can use the ‘--debug’ option to see what is really going on.
When you code an initialization module, be generous in using print statements controlled by the level in ‘$verbose’.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The task of porting DejaGnu is basically that of porting Tcl and
expect
. Tcl and expect
, as distributed with DejaGnu, both
use autoconf
; they should port automatically to most Unix
systems. Currently Tcl and expect
have run on these environments:
Cray running Unicos; HP700 (PA-RISC) running HP-UX; HP300 running HP-UX;
an RS600 running AIX; SGI running IRIX; Sun3 and Sun4 running SunOS;
Sun4 running Solaris 2.x; DECstation running Ultrix; 386BSD; and Linux.
Once Tcl and expect
are ported, DejaGnu should run. Most system
dependencies are taken care of by using expect
as the main
command shell.
[ << ] | [ < ] | [ Up ] | [ > ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Jump to: | -
A B C D E F G H I K L M N O P R S T U V W X Y |
---|
Jump to: | -
A B C D E F G H I K L M N O P R S T U V W X Y |
---|
[Top] | [Contents] | [Index] | [ ? ] |
‘--tool’ selects a particular suite of tests, not the name of the executable program to test. See section Site configuration, for information on the variables that you can use to specify the names of programs to run.
This is also one of the first DejaGnu tests created for GDB. More recent tests use the ‘gdb_test’ procedure, which encapsulates the common pattern of sending a command to GDB, looking for a particular return text, and testing for common failure conditions. Using that procedure, you can write a test equivalent to ‘echo.exp’ simply as ‘ gdb_test "echo Hello world!" "Hello world!" ’
mondfe
is an AMD
specific monitor program freely available from AMD.
mondfe
is specific to
AMD targets using the UDI protocol; it will not work for other targets.
Configuration triples have the form ‘cpu-vendor-os’.
[Top] | [Contents] | [Index] | [ ? ] |
[Top] | [Contents] | [Index] | [ ? ] |
This document was generated on January 16, 2023 using texi2html 5.0.
The buttons in the navigation panels have the following meaning:
Button | Name | Go to | From 1.2.3 go to |
---|---|---|---|
[ << ] | FastBack | Beginning of this chapter or previous chapter | 1 |
[ < ] | Back | Previous section in reading order | 1.2.2 |
[ Up ] | Up | Up section | 1.2 |
[ > ] | Forward | Next section in reading order | 1.2.4 |
[ >> ] | FastForward | Next chapter | 2 |
[Top] | Top | Cover (top) of document | |
[Contents] | Contents | Table of contents | |
[Index] | Index | Index | |
[ ? ] | About | About (help) |
where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:
This document was generated on January 16, 2023 using texi2html 5.0.